Method and apparatus for video error concealment using reference frame selection rules

ABSTRACT

There are provided methods and apparatus for block-based error concealment. An apparatus includes an error concealment module ( 177 ) for concealing a current block in a lost picture in a group of pictures by using a last available inter-coded picture in a previous group of pictures, when the lost picture is a first inter-coded picture in the group of pictures. The group of pictures has a pattern characterized as an intra-coded picture followed by a sequence of inter-coded pictures.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 60/748,250, filed Dec. 7, 2005 and entitled “METHOD AND APPARATUSFOR VIDEO ERROR CONCEALMENT USING REFERENCE FRAME SELECTION RULES,”which is incorporated by reference herein in its entirety.

FIELD OF THE INVENTION

The present invention relates generally to video decoding and, moreparticularly, to a method and apparatus for video error concealmentusing reference frame selection rules.

BACKGROUND OF THE INVENTION

In error-prone transmission environments (e.g., the Internet, wirelessnetworks, and so forth), a transmitted video bitstream may suffercorruptions caused by channel impairment. A common situation encounteredin some practical systems is that certain compressed video frames aredropped from a bitstream. It is especially true for low bit-rateapplications where a frame is small enough to be coded into a transmitunit, such as a real time transport protocol (RTP) packet. At thereceiver end, a robust video decoder should be able to handle the framelosses.

Moreover, when a frame is lost in a corrupted bitstream, frame-basederror concealment methods can be used to conceal the lost frame. Oneprior art approach to frame-based error concealment requires that areference frame that is received and decoded before a detected lostframe be used as the reference frame for the lost frame. According tothe prior art approach, the existence of the previously received anddecoded reference frame is always assumed. However, this practice doesnot necessarily produce the best concealment performance for somespecific frames or bitstream patterns and, thus, may result inunnecessary degradation of concealment quality.

SUMMARY OF THE INVENTION

These and other drawbacks and disadvantages of the prior art areaddressed by the present invention, which is directed to a method andapparatus for video error concealment using reference frame selectionrules.

According to an aspect of the present invention, there is provided anapparatus for block-based error concealment. The apparatus includes anerror concealment module for concealing a current block in a lostpicture in a group of pictures by using a last available inter-codedpicture in a previous group of pictures, when the lost picture is afirst inter-coded picture in the group of pictures. The group ofpictures has a pattern characterized as an intra-coded picture followedby a sequence of inter-coded pictures.

According to another aspect of the present invention, there is provideda method for block-based error concealment. The method includesconcealing a current block in a lost picture in a group of pictures byusing a last available inter-coded picture in a previous group ofpictures, when the lost picture is a first inter-coded picture in thegroup of pictures. The group of pictures has a pattern characterized asan intra-coded picture followed by a sequence of inter-coded pictures.

According to yet another aspect of the present invention, there isprovided an apparatus for block-based error concealment. The apparatusincludes an error concealment module for concealing a current block in alost picture in a group of pictures by using a temporally closestpreceding picture in the group of pictures, irrespective of picturetype, when the lost picture is an inter-coded reference picture. Thegroup of pictures has a pattern characterized as an intra-coded picturefollowed by a sequence of inter-coded reference pictures and inter-codednon-reference pictures.

According to still another aspect of the present invention, there isprovided a method for block-based error concealment. The method includesconcealing a current block in a lost picture in a group of pictures byusing a temporally closest preceding picture in the group of pictures,irrespective of picture type, when the lost picture is an inter-codedreference picture. The group of pictures has a pattern characterized asan intra-coded picture followed by a sequence of inter-coded referencepictures and inter-coded non-reference pictures.

These and other aspects, features and advantages of the presentinvention will become apparent from the following detailed descriptionof exemplary embodiments, which is to be read in connection with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood in accordance with thefollowing exemplary figures, in which:

FIG. 1 illustrates a diagram for an exemplary scalable video decoder towhich the present principles may be applied; and

FIG. 2A and FIG. 2B illustrate a diagram for an exemplary method forframe-based error concealment in a video decoder according to anembodiment of the present principles.

DETAILED DESCRIPTION

The present invention is directed to a method and apparatus for videoerror concealment using reference frame selection rules.

The present description illustrates the principles of the presentinvention. It will thus be appreciated that those skilled in the artwill be able to devise various arrangements that, although notexplicitly described or shown herein, embody the principles of theinvention and are included within its spirit and scope.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the principlesof the invention and the concepts contributed by the inventor tofurthering the art, and are to be construed as being without limitationto such specifically recited examples and conditions.

Moreover, all statements herein reciting principles, aspects, andembodiments of the invention, as well as specific examples thereof, areintended to encompass both structural and functional equivalentsthereof. Additionally, it is intended that such equivalents include bothcurrently known equivalents as well as equivalents developed in thefuture, i.e., any elements developed that perform the same function,regardless of structure.

Thus, for example, it will be appreciated by those skilled in the artthat the block diagrams presented herein represent conceptual views ofillustrative circuitry embodying the principles of the invention.Similarly, it will be appreciated that any flow charts, flow diagrams,state transition diagrams, pseudocode, and the like represent variousprocesses which may be substantially represented in computer readablemedia and so executed by a computer or processor, whether or not suchcomputer or processor is explicitly shown.

The functions of the various elements shown in the figures may beprovided through the use of dedicated hardware as well as hardwarecapable of executing software in association with appropriate software.When provided by a processor, the functions may be provided by a singlededicated processor, by a single shared processor, or by a plurality ofindividual processors, some of which may be shared. Moreover, explicituse of the term “processor” or “controller” should not be construed torefer exclusively to hardware capable of executing software, and mayimplicitly include, without limitation, digital signal processor (“DSP”)hardware, read-only memory (“ROM”) for storing software, random accessmemory (“RAM”), and non-volatile storage.

Other hardware, conventional and/or custom, may also be included.Similarly, any switches shown in the figures are conceptual only. Theirfunction may be carried out through the operation of program logic,through dedicated logic, through the interaction of program control anddedicated logic, or even manually, the particular technique beingselectable by the implementer as more specifically understood from thecontext.

In the claims hereof, any element expressed as a means for performing aspecified function is intended to encompass any way of performing thatfunction including, for example, a) a combination of circuit elementsthat performs that function or b) software in any form, including,therefore, firmware, microcode or the like, combined with appropriatecircuitry for executing that software to perform the function. Theinvention as defined by such claims resides in the fact that thefunctionalities provided by the various recited means are combined andbrought together in the manner which the claims call for. It is thusregarded that any means that can provide those functionalities areequivalent to those shown herein.

Turning to FIG. 1, an exemplary decoder to which the present principlesmay be applied is indicated generally by the reference numeral 100. Thevideo decoder 100 includes an entropy decoder 110. A first output of theentropy decoder 110 is connected in signal communication with an inputof an inverse quantizer/transformer 120. An output of the inversequantizer/transformer 120 is connected in signal communication with afirst input of a summing junction 140.

The output of the summing junction 140 is connected in signalcommunication with a deblock filter 190. An output of the deblock filter190 is connected in signal communication with reference picture stores150. The reference picture store 150 is connected in signalcommunication with a first input of a motion compensator 160. An outputof the motion compensator 160 is connected in signal communication witha second input of the summing junction 140.

A second output of the entropy decoder 110 is connected in signalcommunication with an input of a motion vector (MV) store 176 and asecond input of the motion compensator 160. An output of the motionvector store 176 is connected in signal communication with an input ofan error concealment module 177. An output of the error concealmentmodule 177 is connected in signal communication with a third input ofthe motion compensator 160. The output of the deblock filter 190provides the output of the video decoder 100.

In accordance with the present principles, a number of rules areproposed for the selection of proper frames to be referenced duringerror concealment for lost frames of different types. The followingexemplary cases are considered: (1) in an IPPP . . . bitstream pattern,the first P frame after an I frame is lost; and (2) in an IpPpP . . .bitstream pattern, a P frame is lost. Of course, it is to be appreciatedthat the present principles are not limited to solely the precedingbitstream patterns and, thus, other bitstream patterns may also beutilized in accordance with the present principles. That is, given theteachings of the present principles provided herein, one of ordinaryskill in this and related arts will contemplate these and various otherbitstream patterns to which the present principles may be advantageouslyapplied, while maintaining the scope of the present principles.Moreover, it is to be appreciated that while examples of the presentprinciples are described herein with respect to frames, the presentprinciples are equally applicable to pictures, fields, and so forth.

In the above-described prior art approach, the Joint Model (JM) decodersoftware corresponding to the International Organization forStandardization/International Electrotechnical Commission (ISO/IEC)Moving Picture Experts Group-4 (MPEG-4) Part 10 Advanced Video Coding(AVC) standard/International Telecommunication Union, TelecommunicationSector (ITU-T) H.264 standard (hereinafter the “MPEG4/H.264 standard” orsimply the “H.264 standard”) is equipped with the functionalities todetect and conceal a lost frame. The following two options are availablefor frame loss error concealment in the JM decoder software: (a) framecopy; and (b) motion copy. However, both options require a specificreceived and decoded frame to be referenced during error concealment. Inthe above-described prior art approach, the previously availablereference frame before a lost frame is selected. However, for somespecific bitstream patterns or loss cases, this is not always the bestpractice.

Consider a GOP (group of pictures) with an IPPP . . . bitstream pattern.Suppose the first P frame in the GOP is lost. In accordance with theabove-referenced prior art approach, if “motion copy” error concealmentis invoked to conceal the P frame, the I-frame is selected as itsreference frame. Since the I-frame does not carry motion information(motion vectors are equal to 0), “motion copy” is equivalent to “framerepeat” in this case, which can cause significant concealment distortionand error propagation, especially for high-motion video sequences. AnI-frame is often inserted at a scene change, and in this case the abovepractice generally produces the best possible result. However, anI-frame can also be inserted for other purposes such as error resilienceand random access, where an I-frame does not necessarily correspond to adramatic change in the scene. For the latter case, we propose to use thelast available P frame from the previous GOP as the reference frame forthe concealment. Since the P frame carries motion information, “motioncopy” can be invoked normally to conceal the lost P frame in the currentGOP. Compared to using the I-frame as the reference frame, despite thelarger temporal distance between the two P frames, performanceimprovement can still be achieved for those high-motion video sequencesby utilizing the motion information.

Consider a GOP with an IpPpP . . . bitstream pattern, where “p” denotesa non-reference P frame and “P” denotes a reference P frame. Supposethat a P frame is lost. According to the above-described prior artapproach, its previous available reference frame is used for errorconcealment. In contrast, in accordance with an embodiment of thepresent principles, we propose to use its immediately preceding p frame,if available, for error concealment. Compared to the method in theabove-described prior art approach, the proposed rule uses a frame thatis temporally closer to the lost frame. Therefore, the new referenceframe generally has more correlation to the lost frame, both for pixelvalues and motion information. Hence, “frame repeat” as well as “motioncopy” error concealment methods can benefit from the proposed rule.Since a p-frame is not stored in the decoder buffer, its motioninformation is, thus, not available in the decoder buffer. Therefore,for “motion copy” error concealment, in order to realize the proposedrule, the p-frame in a (P,p) pair has to be stored for the possible lossof the next P-frame. After each time a p-frame is decoded, only thisp-frame needs to be stored and it replaces the existing p-frame in thememory.

Specifically, the decoded pixel values of the p-frame and its motioninformation (motion vectors, reference indices) have to be stored in abuffer allocated for that use. This additional functionality is easy toimplement and does not add much complexity to the decoder.

Turning to FIG. 2A and FIG. 2B, an exemplary method for frame-basederror concealment in a video decoder is indicated generally by thereference numeral 200.

A loop limit block 204 loops over each frame in a sequence, and passescontrol to a decision block 208. The decision block 208 determineswhether or not frame_num(new)−frame_num(old)=1. If so, then control ispassed to a function block 212. Otherwise, control is passed to afunction block 232.

The function block 212 performs normal decoding for a frame, and passescontrol to a function block 216. The function block 216, for an IpP . .. pattern, stores the p frame just decoded so as to replace a previouslydecoded p frame, and passes control to a decision block 220. Thedecision block 220 determines whether or not a frame is ready to output.If so, then control is passed to a function block 224. Otherwise,control is passed to a loop limit block 299.

The function block 224 calculates the picture order count (POC) gap, andpasses control to a decision block 228. The decision block 228determines whether or not the POC gap calculated by function block 224is equal to a POC gap set in a configuration file. If so, then controlis passed to the loop limit block 299. Otherwise, control is passed to afunction block 232.

The loop limit block 299 ends the loop over the frames in the sequence.

The function block 232 creates an empty decoding picture structure for alost frame, and passes control to a decision block 236. The decisionblock 236 determines whether or not frame-copy or motion-copy is to beused. If motion copy is to be used, then control is passed to a functionblock 256. Otherwise, if frame-copy is to be used, then control ispassed to a decision block 240.

The function block 256 checks if the lost frame is the first P-frame ina GOP (so that an adjustment is to be performed for the reference frameof the lost frame), and passes control to a loop limit block 260. Theloop limit block 260 loops over each sub-block in the lost frame, andpasses control to a function block 264. The function block 264 sets themotion vector of a region in a sub-block in the lost frame to that ofthe co-located region in the reference frame, and passes control to afunction block 268. The function block 268 sets the reference index ofthe region to that of the co-located region in the reference frame, andpasses control to a loop limit block 272. The loop limit block 272 endsthe loop over each sub-block in the lost frame, and passes control to afunction block 276. The function block 276 performs motion compensationto reconstruct the frame, and passes control to a function block 280.The function block 280 applies de-blocking filtering to the frame, andpasses control to a decision block 284. The decision block 284determines whether or not the concealed frame is a reference frame. Ifso, then control is passed to a function block 288. Otherwise, controlis passed to a function block 298.

The function block 288 updates frame_num for the concealed frame, andpasses control to a decision block 292. The decision block 292determines whether or not the current frame is an instantaneous decodingrefresh (IDR) frame. If so, then control is passed to a function block294. Otherwise, control is passed to a function block 296.

The function block 294 puts the concealed frame into the decoder buffer,and returns control to the decision block 208.

The function block 296 clears the content in the current decoder buffer,and passes control to the function block 294.

The decision block 240 determines whether or not the reference gap POCis equal to four. If so, then control is passed to a decision block 244.Otherwise, control is passed to a function block 252.

The decision block 244 determines whether or not the current frameincludes an IpP . . . pattern. If so, then control is passed to afunction block 248. Otherwise, control is passed to the function block252.

The function block 248 sets the reference frame to the previousdisposable P frame, and passes control to a function block 252.

The function block 252 performs a pixel-by-pixel copy from the referenceframe to the lost frame, and passes control to the loop limit block 299.

Thus, in accordance with one exemplary embodiment of the presentprinciples, for a GOP with a bitstream pattern characterized as anintra-coded frame followed by a sequence of inter-coded frames (e.g., anIPPPP . . . sequence), when the first inter-coded frame is lost and theintra-coded frame does not correspond to a scene change, the lastavailable predictive frame from the previous GOP is used for errorconcealment.

Moreover, in accordance with another exemplary embodiment of the presentprinciples, for a GOP with a bitstream pattern characterized as anintra-coded frame followed by a sequence of inter-coded frames which areeither of a non-reference type (p) or a reference type (P) (e.g., anIpPpP . . . sequence), when a reference type inter-coded frame is lost(except for the preceding case described with respect to the IPPPP . . .sequence), its temporally closest preceding frame(s) (regardless of theframe type) is used for error concealment.

As noted above, it is to be appreciated that while the presentprinciples are primarily described herein with respect to frames, thepresent principles are equally applicable to pictures, fields, and soforth, as is readily ascertained by one of ordinary skill in this andrelated arts. Moreover, it is to be appreciated that the presentprinciples are not limited to solely the preceding bitstream patternsand, thus, other bitstream patterns may also be utilized in accordancewith the present principles. That is, given the teachings of the presentprinciples provided herein, one of ordinary skill in this and relatedarts will contemplate these and various other bitstream patterns towhich the present principles may be advantageously applied, whilemaintaining the scope of the present principles.

A description will now be given of some of the many attendantadvantages/features of the present invention, some of which have beenmentioned above. For example, one advantage/feature is an apparatus forblock-based error concealment that includes an error concealment modulefor concealing a current block in a lost picture in a group of picturesby using a last available inter-coded picture in a previous group ofpictures, when the lost picture is a first inter-coded picture in thegroup of pictures. The group of pictures has a pattern characterized asan intra-coded picture followed by a sequence of inter-coded pictures.

Another advantage/feature is the apparatus as described above, whereinthe error concealment module conceals the current block by derivingmotion information for the current block from a co-located block in thelast available predictive picture.

Yet another advantage/feature is the apparatus having the errorconcealment module that derives the motion information for the currentblock from a co-located block as described above, wherein the errorconcealment module derives a reference index for use for the currentblock from a reference index of the co-located block.

Moreover, another advantage/feature is the apparatus as described above,wherein the intra-coded picture in the group of pictures is withoutcorrespondence to a scene change.

Further, another advantage/feature is the apparatus as described above,wherein the error concealment module performs a pixel-by-pixel copy froma co-located pixel in the last available predictive picture to a currentpixel in the lost picture, when a picture order count gap from the lostpicture in the group of pictures to the last available predictivepicture in the previous group of pictures is greater than a threshold.

Also, another advantage/feature is the apparatus as described above,wherein the error concealment module updates a picture order count ofthe lost picture.

Additionally, another advantage/feature is an apparatus for block-basederror concealment that includes an error concealment module forconcealing a current block in a lost picture in a group of pictures byusing a temporally closest preceding picture in the group of pictures,irrespective of picture type, when the lost picture is an inter-codedreference picture. The group of pictures has a pattern characterized asan intra-coded picture followed by a sequence of inter-coded referencepictures and inter-coded non-reference pictures.

Another advantage/feature is the apparatus having the error concealmentmodule that conceals the current block using the temporally closestpreceding picture as described above, wherein the error concealmentmodule conceals the current block by deriving motion information for thecurrent block from a co-located block in the temporally closestpreceding picture.

Yet another advantage/feature is the apparatus having the errorconcealment module that conceals the current block using the temporallyclosest preceding picture and that derives the motion information forthe current block from the co-located block as described above, whereinthe error concealment module derives a reference index for use for thecurrent block from a reference index of the co-located block.

Moreover, another advantage/feature is the apparatus having the errorconcealment module that conceals the current block using the temporallyclosest preceding picture as described above, wherein the intra-codedpicture in the group of pictures is without correspondence to a scenechange.

Further, another advantage/feature is the apparatus having the errorconcealment module that conceals the current block using the temporallyclosest preceding picture as described above, wherein the errorconcealment module performs a pixel-by-pixel copy from a co-locatedpixel in the temporally closest preceding picture to a current pixel inthe lost picture, when a picture order count gap from the lost pictureto the temporally closest preceding picture is greater than a threshold.

Also, another advantage/feature is the apparatus having the errorconcealment module that conceals the current block using the temporallyclosest preceding picture as described above, wherein the errorconcealment module updates a picture number of the lost picture.

Also, another advantage/feature is the apparatus having the errorconcealment module that conceals the current block using the temporallyclosest preceding picture as described above, wherein the apparatusfurther includes a buffer for storing the lost picture subsequent toconcealment, when the lost picture is an instantaneous decoding refreshpicture.

Additionally, another advantage/feature is the apparatus having theerror concealment module that conceals the current block using thetemporally closest preceding picture as described above, wherein thelost picture is stored so as to replace a previously concealed lostpicture.

Moreover, another advantage/feature is the apparatus having the errorconcealment module that conceals the current block using the temporallyclosest preceding picture as described above, wherein the apparatusfurther includes a buffer for storing the last available non-referenceinter-coded picture for use in concealing the current block in the lostpicture, subsequent to decoding of the last available non-referenceinter-coded picture.

Further, another advantage/feature is the apparatus having the errorconcealment module that conceals the current block using the temporallyclosest preceding picture as described above, wherein the apparatusfurther includes a buffer for storing motion vectors for the lastavailable non-reference inter-coded picture for use in concealing thecurrent block in the lost picture, subsequent to decoding of the lastavailable non-reference inter-coded picture.

These and other features and advantages of the present invention may bereadily ascertained by one of ordinary skill in the pertinent art basedon the teachings herein. It is to be understood that the teachings ofthe present invention may be implemented in various forms of hardware,software, firmware, special purpose processors, or combinations thereof.

Most preferably, the teachings of the present invention are implementedas a combination of hardware and software. Moreover, the software may beimplemented as an application program tangibly embodied on a programstorage unit. The application program may be uploaded to, and executedby, a machine comprising any suitable architecture. Preferably, themachine is implemented on a computer platform having hardware such asone or more central processing units (“CPU”), a random access memory(“RAM”), and input/output (“I/O”) interfaces. The computer platform mayalso include an operating system and microinstruction code. The variousprocesses and functions described herein may be either part of themicroinstruction code or part of the application program, or anycombination thereof, which may be executed by a CPU. In addition,various other peripheral units may be connected to the computer platformsuch as an additional data storage unit and a printing unit.

It is to be further understood that, because some of the constituentsystem components and methods depicted in the accompanying drawings arepreferably implemented in software, the actual connections between thesystem components or the process function blocks may differ dependingupon the manner in which the present invention is programmed. Given theteachings herein, one of ordinary skill in the pertinent art will beable to contemplate these and similar implementations or configurationsof the present invention.

Although the illustrative embodiments have been described herein withreference to the accompanying drawings, it is to be understood that thepresent invention is not limited to those precise embodiments, and thatvarious changes and modifications may be effected therein by one ofordinary skill in the pertinent art without departing from the scope orspirit of the present invention. All such changes and modifications areintended to be included within the scope of the present invention as setforth in the appended claims.

1. An apparatus for block-based error concealment, comprising: an error concealment module for concealing a current block in a lost picture in a group of pictures by using a last available inter-coded picture in a previous group of pictures, when the lost picture is a first inter-coded picture in the group of pictures, the group of pictures having a pattern characterized as an intra-coded picture followed by a sequence of inter-coded pictures.
 2. The apparatus of claim 1, wherein said error concealment module conceals the current block by deriving motion information for the current block from a co-located block in the last available predictive picture.
 3. The apparatus of claim 2, wherein said error concealment module derives a reference index for use for the current block from a reference index of the co-located block.
 4. The apparatus of claim 1, wherein the intra-coded picture in the group of pictures is without correspondence to a scene change.
 5. The apparatus of claim 1, wherein said error concealment module performs a pixel-by-pixel copy from a co-located pixel in the last available predictive picture to a current pixel in the lost picture, when a picture order count gap from the lost picture in the group of pictures to the last available predictive picture in the previous group of pictures is greater than a threshold.
 6. The apparatus of claim 1, wherein said error concealment module updates a picture order count of the lost picture.
 7. A method for block-based error concealment, comprising: concealing a current block in a lost picture in a group of pictures by using a last available inter-coded picture in a previous group of pictures, when the lost picture is a first inter-coded picture in the group of pictures, the group of pictures having a pattern characterized as an intra-coded picture followed by a sequence of inter-coded pictures.
 8. The method of claim 7, wherein said concealing comprises deriving motion information for the current block from a co-located block in the last available predictive picture.
 9. The method of claim 8, wherein said concealing comprises deriving a reference index for use for the current block from a reference index of the co-located block.
 10. The method of claim 7, wherein the intra-coded picture in the group of pictures is without correspondence to a scene change.
 11. The method of claim 7, wherein said concealing comprises performing a pixel-by-pixel copy from a co-located pixel in the last available predictive picture to a current pixel in the lost picture, when a picture order count gap from the lost picture in the group of pictures to the last available predictive picture in the previous group of pictures is greater than a threshold.
 12. The method of claim 7, wherein said concealing comprises updating a picture order count of the lost picture. 13-32. (canceled) 